Method AND APPARATUS FOR IMAGE PROCESSING 

BACKGROUND OF THE INVENTION 

[0001] 1. Field of the Invention 

[0002] The present invention relates to an image processing apparatus built into a 
communication device (for example, a TV phone, a TV conference system, a video mail 
system, a video chat system, an intercom, etc.), handling unage data, more particularly 
relates to an image processing apparatus editmg a personal image (for example, a face 
image of a user, a whole-body image of the user, etc.). 
[0003] 2. Description of flie Related Art 

[0004] There is an amusement apparatus that prints and outputs a personal image of the 
user to various kinds of print media. The amusement apparatus laps a frame of 
omaments (for example, a picture, a pattem, etc.) over the personal image to malce a 
composite image, and prints and outputs the composite image to a print media (for 
example, seal etc.). 

[0005] The amusement apparatus does not have a function to adjust a positional 
relationship of the personal image and the omaments, in itself. Therefore, even if the 
user has arranged the personal image to fit in the ornament frame, the omament may 
overlap with the personal image, so that the omament hides a face part of the user 
and/or a head part of the user and the personal image becomes unclear. 
[0006] Considering this point, a reference 1 (pubUshed Japanese Patent Application 
Laid-Open No. 2000-22929) discloses a technique that adds omaments to a personal 
image, while maldng an omament not overlap with a body part of the personal unage, 
especially, a predetermined body part corresponding to a face and heads. 
[0007] Recently, a camera-built-in cellular phone is put in practical use. The cellular 
phone has a function to lap a frame of omaments over a personal image to make a 
composite image and a function to transmit the composite image. If a transmitter uses 
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such a camera-built-in cellular phone, a receiver can enjoy seeing the picture that the 
transmitter has edited. 

[0008] Reference 1 assumes that ornaments do not move. According to the technique 
disclosed by reference 1, omaments are added to a personal image such that the 
ornaments do not overlap with a body part of the personal image, especially, a 
predetermined body part area corresponding to a face and heads. 

[0009] Consideration of reference 1 may be appropriate, since, in reference 1, output is 
designated only to a printer that prints the composite image to a print media and an 
image once printed on the print media is rarely edited again. 

[0010] However, the camera-built-in cellular phones mentioned above have spread. In 
one of these cellular phones, a composite image is displayed on a display, without being 
printed to a print media. In the future, the user of one of the above-mentioned cellular 
phones will think that he/she wants to move omaments in order to obtain more pleasure. 
[0011] However, in prior art, adding omaments which do not move to a personal image 
which does not move as well is only taken into consideration. Adding omaments which 
move to the personal image is not taken into consideration at all. For this reason, if prior 
art is simply appUed to a personal image which moves, the personal image and the 
omaments may interfere with each other. As a result, the composite image becomes 
unclear and does not look more interesting. 
OBJECTS AND SUMMARY OF THE INVENTION 

[0012] An object of the present invention is to provide an image processing method, by 
which a personal image can be displayed more clearly, making the image look more 
interesting, and an art related thereto. 

[0013] A first aspect of the present invention provides an image processing method 
comprising: detecting a position of a body part area in an input image; locating a 
coordinate system of an ornament according to the detected position of the body part 
area in the input image; and ou^utting an ornament-arranged input image, based on the 
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located coordinate system of the ornament. 

[0014] According to the method described above, when a person photographic object 
moves in an input image, an omament can be moved according to the movement of the 

person photographic object. 

[0015] A second aspect of the present invention provides an image processing method 
as defined in the first aspect, wherein the outputting comprises ou^utting an image data 
of the omament-drawn input image. 

[0016] According to the method described above, it is not necessary to compose the 
input image and the omament in a destination to which the image is outputted. 
[0017] A third aspect of the present invention provides an image processing method as 
defined in the first aspect, wherein the outputting comprises outputting an image data 
relating the omament to the input image. 

[0018] According to the method described above, it is easy to restore the input image in 
a destination to which the image is outputted, and it is convenient. 
[0019] A fourth aspect of the present invention provides an image processing method as 
defined in the third aspect, wherein the omament and the input image are related using a 
meta file. 

[0020] According to the method described above, it is possible to relate the input image 
to the omament concisely. 

[0021] A fifth aspect of the present invention provides an image processing method as 
defined in the foiuth aspect, wherein a form of the meta file is SMIL. 
[0022] According to the method described above, relating the input image to the 
omament can be easily performed. 

[0023] A sixth aspect of the present invention provides an image processing method 
comprising: detecting a size of a body part area in an input image; scaling a coordinate 
system of an omament according to the detected size of the body part area; and 
outputting an omament-arranged input image, based on the scaled coordinate system of 
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the ornament. 

[0024] According to the method described above, when a person photographic object 
gets closer to an image pick-up device, both size and movement path of the omament 
become larger. On the other hand, when the person photographic object moves away 
from the image pick-up device, both size and movement path of the omament become 
smaller. Thereby, it is possible to make the omament look like following the person 
photographic object that moves back and forth. 

[0025] A seventh aspect of the present invention provides an image processing method 
comprising: detecting a rotation amount of a body part area, the rotation being 
in-plane of an input image; defining a rotation amount of a coordinate system of an 
omament according to the detected rotation amount of the body part area; and 
outputting an ornament-arranged input image, based on the defined rotation amount of 
the coordinate system of the omament. 

[0026] According to the method described above, when a face is leaned aside, the 
omament itself and the movement path of the omament can be leaned correspondingly. 
Therefore, it is possible to make the omament look like following the person 
photographic object that rotates in-plane of the input image. 

[0027] An eighth aspect of the present invention provides an image processing method 
comprising: detecting a rotation amount of a body part area, the rotation being 
off-plane of an input image; defining a rotation amount of a coordinate system of an 
omament according to the detected rotation amount of the body part area; and 
outputting an ornament-arranged input image, based on the defined rotation amount of 
the coordinate system of the omament. 

[0028] According to the method described above, when a person does a "right-about", 
the omament and the movement path of the omament can be transformed according to 
the rotation of the person. Therefore, it is possible to make the omament look like 
following the person photographic object that rotates off-plane of the input image. 
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[0029] A ninth aspect of the present invention provides an image processing method 
comprising: detecting a body part area in an input image; defining opacity of an 
ornament, the opacity of the ornament in a foreground area with respect to the detected 
body part area being different fi-om the opacity of the ornament in a background area of 
the ornament with respect to the detected body part area; and outputting an 
ornament-arranged input image, based on the defined opacity of the ornament. 
[0030] According to the method described above, when the movement path of the 
omament comes to the back of the body part area, the omament can be hidden. 
Therefore, it is possible to make the omament look like surrounding the body part area 
and moving in three directions about the body part area. 

[0031] A tenth aspect of the present invention provides an image processing method as 
defined in the first aspect, wherein, when an omament collides with a body part area, a 
moving direction of the omament is altered so as to move the omament away from the 
body part area. 

[0032] According to the method described above, a motion like reflection can be given 
to the omament, when the omament collides with the body part area. 
[0033] An eleventh aspect of the present invention provides an image processing 
method as defined in the first aspect, wherein a body part area is a face part area of a 
person as a photographic object. 

[0034] In the method described above, an image in which a face area is chosen as the 
body part area can be more interesting, since the face vividly shows the person's 
intention and feeling. 

[0035] The above, and other objects, features and advantages of the present invention 
will become apparent from the following description read in conjunction with the 
accompanying drawings, in which like reference numerals designate the same elements. 
BRIEF DESCRIPTION OF THE DRAWINGS 

[0036] Fig. 1 is a functional block diagram, illustrating how an image processing 
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apparatus functions according to the first embodiment of the present invention; 

[0037] Fig. 2 is a block diagram, illustrating the image processing apparatus according 

to the first embodiment of the present invention; 

[0038] Fig. 3 is an illustration, showing an outside view of the image processing 
apparatus according to the first embodiment of the present invention; 
[0039] Fig. 4 (a) is an illustration, showing a meta file according to the first 
embodiment of the present invention; 

[0040] Fig. 4 (b) is an illustration, showing an input image file according to the first 
embodiment of the present invention; 

[0041] Fig. 4 (c) is an illustration, showing an omament file according to the first 
embodiment of the present invention; 

[0042] Fig. 5 is a flowchart, illustrating the image processing apparatus according to the 
first embodiment of the present invention; 

[0043] Figs. 6 (a) to (c) are illustrations, showing templates according to the first 
embodiment of the present invention; 

[0044] Figs. 7 (a) to (c) are illustrations, showing the templates according to the first 
embodiment of the present invention; 

[0045] Figs. 8 (a) to (c) are illustrations, showing the templates according to the first 
embodiment of the present invention; 

[0046] Fig. 9 is an explanatory diagram, illustrating pattem matching according to the 
first embodiment of the present invention; 

[0047] Figs. 10 (a) and (b) are block diagrams, illustrating the omament information 
according to the first embodiment of the present invention; 

[0048] Figs. 11 (a) to (c) are illustrations, showing three kinds of display of a user 
interface according to the first embodiment of the present invention; 
[0049] Figs. 12 (a) to (g) are explanatory diagrams, showing a process of the image 
processing apparatus according to the first embodiment of the present invention; 
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[0050] Fig. 13 is an explanatory diagram, showing a coordinate system according to the 
first embodiment of the present invention; 

[0051] Figs. 14 (a) to (d) are explanatory diagrams, showing an animation according to 
the first embodiment of the present invention; 

[0052] Fig. 15 is an explanatory diagram, showing the animation according to the first 
embodiment of the present invention; 

[0053] Fig. 16 (a) is an illustration, showing an omament according to the first 
embodiment of the present invention; 

[0054] Fig. 16 05) is an explanatory diagram, showing an anchor point according to the 
fiirst embodiment of the present invention; 

[0055] Fig. 16 (c) is an illustration, showing a composite image according to the first 
embodiment of the present invention; 

[0056] Fig. 17 is an illustration, showing the omament according to the first 
embodiment of the present invention; 

[0057] Fig. 18 is an illustration, showing the composite image according to the first 
embodiment of the present invention; 

[0058] Fig. 19 is an illustration, showing an omament according to the first embodiment 
of the present invention; and 

[0059] Fig. 20 is an illustration, showmg the composite image according to the first 
embodiment of the present invention. 

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS 

[0060] Hereinafter, a description is given of embodiments of the invention with 

reference to the accompanying drawings. 

[0061] Prior to detailed description of constructions, important terms are explained. 
[0062] A "personal image" is an image containing a part or a whole image of a person. 
Therefore, the image may be a whole-body image, a face image, an image of a sight of 
the person's back, or the upper half of the body. The image may also be taken with two 
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or more people. Any kind of patterns, such as scenery and a design other than a person 
may comprise a backgroimd. 

[0063] A "body part" means a part of a person's body. The part of the person's body is 
included in the body part, when the part can be recognized as the part of the person's 
body, even when the body part is invisible because the skin of the body part is equipped 
with dress, a hat, or shoes. Therefore, a face is a body part and a head is also a body part. 
An eye, a nose, a mouth, an ear, eyebrows, hair, a neck, the upper half of the body with 
dress, a hand, an arm, a leg, feet with shoes, a head with a hat, eyes with glasses are also 
body parts. 

[0064] A "body part area" is an area defined as an area which the body part occupies in 
the personal image. The body part area may include, within itself, a part that is not the 
body part, and may be an area that is located inside the body part, 

[0065] Suppose that the body part is a face, the body part area may include an area in 
the circumference of the face, or the body part area may be a minimum-sized rectangle 
surrounding eyes, a mouth, and a nose, but not containing a forehead and ears. 
[0066] An "omament" is an image pattern to be added. The image pattem may be an 
image pattem that is stored in advance, or may be an image pattem generated with 
computer graphics technology. The image pattem may be a pattem of a character, a 
pattem of a symbol, and a pattem of a figure. 

[0067] Fig. 1 is a block diagram, illustrating how the image processmg apparatus 
functions according to the first embodiment of the present invention. As shown in Fig. 1, 
the image processing apparatus of the present embodiment has the following 
components. 

[0068] An image input unit 1 acquires an input image. In the present embodiment, the 
input image contains a personal image which is permitted to move. The input image 
generally means an image for one frame of a moving picture. 

[0069] The input image may be an image data immediately after photographed with a 
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camera, encoded data according to coding methods, such as MPEG, after photographing 
(the encoded data could be loaded from a recording medium or received from a 
communication device), or an image data obtained by decoding the encoded data. 
[0070] A control unit 2 controls each element shown in Fig. 1 according to the flow 
chart of Fig. 5. 

[0071] An image output unit 3 outputs an image that is composite of the input image 
and the omament. The output image may be displayed on a display device, or may be 
encoded according to the above-mentioned coding method, and stored in a recording 
medium, or may be transmitted by other conununication devices. 

[0072] An input image storing unit 4 stores an input image that the image input unit 1 
inputted. 

[0073] A template storing unit 5 stores templates of a body part area. Hereinafter, a face 
area is regarded as a body part area in the present embodiment. 

[0074] As shown in Figs. 6 (a), (b), and (c), templates made by each modeled outline of 
face parts (a head, eyes, a nose, and a mouth) are prepared in different sizes for use. 
[0075] As shown in Figs. 7 (b) and (c), templates with a certain in-plane rotation 
amount of an input image are used, besides the template of a basic position as shown in 
Fig. 7 (a). Thereby, the inclination of the face by rotation can be detected. 
[0076] As shown in Fig. 8 (b) and (c), templates with a certain off-plane rotation 
amount of an input image are used, besides the template of a basic position as shown in 
Fig. 8 (a). Thereby, the direction of the face can be detected when the direction of the 
face differs from the basic position (when the normal vector passing through the center 
of the face is not orthogonal to the plane of an input image). 

[0077] Using the templates of the template storing unit 5, a detecting unit 6 shown in 
Fig. 1 detects a body part area from the input image that is stored by the input image 
storing unit 4. 

[0078] As shown in Fig. 9, the detecting unit 6 extracts an edge component, filtering the 
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input image stored in the input image storing unit 4 with a differential filter. Moreover, 
the detecting unit 6 selects a template 53 from the template storing unit 5, and performs 
pattern matching using the selected template 53 and the edge component. 
[0079] The pattern matching is a processing which moves the template 53 (Nx * Ny 
pixels) over a search area (Mx - Nx +1) (My - Ny + 1) in an input image (Mx * My 
pixels) which is larger than the template, and searches the upper left position (a, b) of 
the template at which a residual R given by the following equation becomes minimum. 
Here, the sjonbol indicates multiplication. 
[0080] [Equation 1] 



[0081] In Equation 1, 1 (a, b) (mx, my) is a partial image of an input image, and T (mx, 
my) is an image of the template 53. 

[0082] Instead of Equation 1, the upper left position (a, b) of the template may be 
searched by calculating a cross-correlation coefficient C defined by Equation 2, and 
finding a maximum value for the cross-correlation coefficient C. 
[0083] [Equation 2] 
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[0084] The detecting unit 6 uses various kinds of templates shown in Fig. 6, Fig. 7, and 
Fig. 8. When the detecting unit 6 uses Equation 1, the detecting unit 6 finds a template 
with which the residual R becomes minimum. When the detecting unit 6 uses Equation 
2, the detecting unit 6 finds a template with which the correlation coefficient C becomes 
maximum. The detecting unit 6 regards the values of position, size, inclination, and 
direction of the matched template as the values of position, size, inclination, and 
direction of the face image of the input image. 

[0085] In the case of a face image, the transition of the position, the size, the inclination, 
and the direction of the face may be obtained by performing the same processing for 
every frame or every several frames. 

[0086] When more than one face exist in the input image, the detecting unit 6 detects, as 
the face part area of a second person, a position of the lowest residual R or the highest 
correlation coefficient C within the input image area from which the already detected 
face part area as the first person is excluded. 

[0087] Similarly, for a face after a third person and on, the detecting unit 6 detects 
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repeatedly, while setting up the low position of the residual R and the high position of 
the correlation coefRcient C as the face part area in the area except for the detected face 
part areas, until the residual R becomes larger than a previously defined threshold or 
until the correlation coefficient C becomes smaller than a previously defined threshold. 
Thereby, face part area for two or more people is detectable. 

[0088] The above-mentioned detection need not always be performed for all the values. 
The detection of values except for the position and size of the face area may be optional. 
[0089] In Fig. 1, an omament information storing unit 7 stores omament information. 
The omament information is explained using Fig. 10. 

[0090] In an example (omament information no. 1) shown in Fig. 10 (a), the omament 
information consists of the omament image data and a transform property, the omament 
image data being related to the transform property. 

[0091] In the field of the "omament image data", the omament image data itself or an 
ID or a pointer which indicates the omament image data itself is stored. As illustrated in 
Fig. 10 (a), the omament image data may consist of only the image data, or have 
additional attributes (ex. behavior in a collision etc.). 

[0092] Stored in the field of the "transform property" is information how the omament 
is moved with regard to the omament layer and how the omament layer is overlapped 
on the input image. 

[0093] Specifically, the omament is defined on a local coordinate system (let an anchor 
point be the origin) by the transform property, the local coordinate system being 
different from the coordinate system of the input image. The local coordinate system 
has attributes of a position, a scale (rate of scaling), rotation, and opacity. 
[0094] In more detail, the anchor point shows a position and a direction of the origin of 
the local coordinate system concerning the omament. The values set to fields of 
"position", "scale", and "rotation" indicate the position, the scale (rate of scaling), and 
the rotational degree of the actual image on the coordinate system that the anchor point 
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has (that is the local coordinate system). 

[0095] In an example (ornament information no. 2) shown in Fig. 10 (b), the ornament 
information consists of the ornament image data and a mask data, the omament image 
data being related to the mask data. 

[0096] In this case, the "omament image data" is data displayed as a background, and 
the "mask data'' indicates a size and positional relationship of the data part of the 
backgroimd (shaded area) and the part in which the image of the face area appears 
(unshaded area). 

[0097] In Fig. 1, an omament arranging unit 8 defines the arrangement information of 
the omament in accordance with the change of the body part area which the detecting 
unit 6 has detected. 

[0098] When a maximum matching is obtained with one of the templates shown in Fig. 

7, the detecting unit 6 detects the rotation amount of the body part area, the rotation 
being in-plane of the input image; and the omament arranging unit 8 defines the 
arrangement information of the omament in accordance with the change of the rotation 
amount that the detecting unit 6 has detected. 

[0099] When a maximum matching is obtained with one of the templates shown in Fig. 

8, the detecting unit 6 detects the rotation amoimt of the body part area, the rotation 
being off-plane of the input image; and the omament arranging unit 8 defines the 
arrangement information of the omament in accordance with the change of the rotation 
amount which the detecting unit 6 has detected. 

[0100] A detailed processing of the omament arranging imit 8 is explained later, citing 
several kinds of examples. 

[0101] In Fig. 1, a composing unit 10 outputs an omament-arranged input image, in 
which an omament is arranged in the input image based on the arrangement information 
that the omament arrangement unit 8 has defined. The omament-arranged input image 
is once stored in an ou^ut image storing unit 11, and outputted from the image output 
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units. 

[0102] In order to perform displaying on a display device, the composing unit 10 
generates image data in which an ornament is directly drawn to the input image (data 
form 1). As an image data for transmitting to a recording medium or to other 
communication devices, the composing unit 10 may preferably generate and transmit 
image data that an input image is related to parameters in drawing the ornament (data 
form 2). According to the data form 2, the input image can be easily restored, therefore 
it is advantageous. An explanation for the data form 2 will be supplied later. 
[0103] In Fig. 1, an operation unit 12 is for a user to input operation information into the 
control unit 2. 

[0104] Next, construction examples for the image processing device shown in Fig. 1 
and the pertaining peripheral parts are explained using Fig. 2. Fig. 2 is a block diagram 
of the above-mentioned image processing apparatus. 

[0105] The image processing apparatus shown in Fig. 2 exemplifies a camera-built-in 
cellular phone, in which the image processing apparatus shown in Fig. 1 is installed. As 
shown in Fig. 2, the camera-built-in cellular phone has the following elements. 
[0106] CPU 21 controls each element of Fig. 2 via a bus 20, and executes the control 
program that is recorded on ROM 23 and follows the flowchart as shown in Fig. 3. 
[0107] RAM 22 secures a temporary storage area that CPU 21 requires for processing. 
A flash memory 24 is a device equivalent to a recording medium. 

[0108] Moreover, a communication processing unit 26 performs transmission and 
reception of data with an external communication device via an antenna 25. 
[0109] An image processing unit 27 consists of an encoder/a decoder for a coding 
system, such as MPEG, processes the image that a camera 28 has photographed, and 
controls the display state of LCD 29 (an example of a display device) based on the 
image data that CPU 21 has directed. 

[0110] An audio processing unit 30 controls an audio input from a microphone 31, and 
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an audio output via a speaker 32. 

[0111] An interface 33 is connected to the bus 20. A user can input operation 
information by a key set 34 via the interface 33, and can connect the other devices via a 
port 35. 

[0112] The function of an image input unit 1 is realized by a process in which CPU 21 
or the image processing unit 27 performs operation for data recorded in the flash 
memory 24 or data obtained by photographing with the camera 28. 
[0113] The functions of the control unit 2, the detecting unit 6, the ornament arranging 
unit 8 and the composing unit 10 are realized by processes in which CPU 21 exchanges 
data with RAM 22, the flash memory 24 and others, and in which CPU 21 performs 
operation for the data. 

[0114] Furthermore, the input image storing unit 4, the template storing unit 5, the 
omament information storing unit 7, an arrangement information storing unit 9, and the 
output image storing unit 11 correspond to the storage areas secured by RAM 22, ROM 
23 or the flash memory 24. The control unit 12 of Fig. 1 corresponds to the key set 34 of 
Fig. 2. 

[0115] CPU 21 performs recognition of the operation that a user performs on the key set 
34, acquisition of the image from the camera 28, compression of a camera image and 
storage to the flash memory 24, loading and expansion of the storage image, image 
composition, image reproduction, and display on LCD 29 and so on. The image 
processing unit 27 may perform some items of the above-described processing. 
[0116] Fig. 3 indicates the outside view of the image processing apparatus according to 
the present embodiment. As mentioned above, the image processing apparatus 
according to the present embodiment is installed in the camera-built-in cellular phone. 
[0117] The camera-built-in cellular phone of Fig. 3 is a flip-open type, which an upper 
casing 42 and a lower casing 41 are connected foldably by a joint portion 43. Needless 
to say, the appearance of the image processing apparatus is arbitrary, and the present 
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invention can be applied to devices such as PDA other than cellular phones. 
[0118] Explanation of the data form 2 is supplemented referring to Fig. 4. Assume that 
an ornament (file name "star.swf ') shown in Fig. 4 (c) is to be related with an input 
image (file name "face.rm") shown in Fig. 4 (b). 

[0119] In this case, it is desirable to use the meta file in SMIL (Synchronized 
Multimedia Integration Language), as shown in Fig. 4 (a). In Fig. 4 (a), a line <video 
src="face.rm"> and a line <animation src="star.swf '>, which are held between <par> 
and </par>, are simultaneously executed. After all, the input image of Fig. 4 (b) and the 
omament of Fig. 4 (c) are composed at the time of a display. 

[0120] In establishing relationship such as shown in Fig. 4, it is not necessary to create a 
composite image itself of the input image of Fig. 4 (b) and the omament of Fig. 4 (c). 
[0121] Not only can the metafile in SMIL be used as described above, but also a file in 
XML (Extensible Markup Language) or an MP4 file and so on may be used. When the 
data form 2 is used, it is sufficient that the input image and the omament are related by a 
certain form. 

[0122] Next, the flow of processing by the image processing apparatus according to the 
present embodiment is ouflined using Fig. 1, Fig. 5, and Fig. 12. It is assumed that the 
omament information (part 1) indicated in Fig. 10 (a) is used. 

[0123] In Step 1, the control unit 2 confirms whether the processing has completed or 
not regarding all the frames that constitute a moving image. When it has completed, the 
processing is ended. When it is not completed, the processing moves to Step 2. 
[0124] In Step 2, the image input unit 1 acquires the input image as shown in Fig. 12 (a), 
and the control unit 2 saves the input image in the input image storing unit 4. 
[0125] In Step 3, the control unit 2 orders the detecting unit 6 to perform detection for 
the face area. Thereby, the detecting unit 6 detects the face area using the template (Fig. 
12 (b)) of the template storing unit 5. 

[0126] In Step 4, the control unit 2 orders the omament arrangement unit 8 to determine 
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arrangement information. Thereby, the ornament arrangement unit 8 uses the omament 
information of the omament information storing unit 7, defines arrangement 
information, and saves the arrangement information in the arrangement information 
storing unit 9. Consequentiy, the omament image on the omament layer as shown in Fig. 
12 (c) is acquired. 

[0127] In Step 5, the control unit 2 orders the composing unit 10 to perform 
composition of the input image and the omament. Thereby, according to the omament 
information saved in the omament information storing unit 7, as shown in Fig. 12 (d), 
the composing unit 10 composes the omament in the input image, and saves the 
resultant composite image in the output image storing unit 11. 

[0128] In Step 6, the control unit 2 outputs the composite image of the output image 
storing unit 11 to the exterior via the image output unit 3. 

[0129] The control unit 2 repeats steps after Step 1 until the processing of all frames is 
completed. 

[0130] Next, an example of composition of a user interface is explained using Fig. 9. 
Here, the application chosen by the user is supposed to be image edit software, and is 
also supposed to be image composition mode that composes the omament and the 
personal image photographed by the user. 

[0131] First, as shown in Fig. 11 (a), a personal image as the object to be edited is 
displayed on the LCD 29. Attaching to the personal image, an edit button 40 and a 
return button 41 are displayed. 

[0132] When a user chooses the edit button 40 with the key set 34, the list of the 
omament that can be inserted is displayed, as shown in Fig. 11 (b). 
[0133] When the user chooses the item "tears" and pushes a determination button 43, 
the face area of the personal image will be detected, the omament arrangement 
information suitable for the face area will be searched for, the omament "tears" will be 
added to the input picture according to the omament arrangement information, and the 
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result will be displayed on the LCD 29, as shown in Fig. 11 (c). In Fig. 11 (c), the 
ornament of tears 50 and 51 are added to the input picture. Here, the tears are inserted as 
moving animation as well as a still image. 

[0134] In the prior art, tears do not move and stay still where the tears are arranged. 
[0135] Even if animation is inserted, the movement of the teas in the animation is 
completely irrelevant to the movement of the personal image. 

[0136] For example, in the prior art, tears keep running downward (even under the face), 
because tears are not automatically inserted right below the eyes, and tears are not 
automatically adjusted to stop running around the cheeks of the face. Moreover, when 
the face inclines or moves, the motion of the tears carmot be modified in accordance 
with the face. 

[0137] On the other hand, according to the present embodiment, tears are designed to be 
automatically inserted right below the eyes, synchronizing with the movement of the 
body part area, because the location and size of the body part area (for example, the 
eyes within the face area in the present embodiment) are detected before tears are 
inserted. Moreover, the tears are able to follow the movement of the face. 
[0138] Hereinafter, the ornament arrangement information is further explained, while 
exhibiting application examples according to the present embodiment. Fig. 13 is an 
explanatory diagram, illustrating an input image and a positional relationship of a 
coordinate system for the omament. 

[0139] A procedure that creates general animation is briefly explained as supplement 
before explaining the application examples. Basically, the animation, in which the 
omament moves, is created by moving an omament layer that is related to the transform 
property. 

[0140] Hence, in order to create animation, the attributes (position, rotation, etc.) of the 
transform are set to every key frame (existence located on a time-axis), and each value 
of the attributes is changed according to the passage of time. Here, the key frame means 
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a frame to which the values of the position and other properties of the layer at a time of 
criterion are attributed. When there are two key frames (their positions on time axe 
naturally differ) that have different attribute values, for any frame locating between the 
key frames in the time axes, each value of attributes can be determined by calculating a 
mean value regarding the frame. In short, animation can be created by repeating the 
operations of (1) creating the key frames and (2) setting up the move locus by 
interpolation between the key frames. 

[0141] Next, the procedure of creating animation by the above-mentioned technique is 
explained, using the animation shown in Fig. 14 as the example. Here, as shown in Figs. 
14 (a) to (d), animation, in which an airplane flies from lower left side to upper right 
side, is created by changing the location, rotation, and scale of the layer. 
[0142] In order to define the coordinate system that arranges an omament layer, the 
anchor point of the coordinate system is set up first. In the example, as indicated in Fig. 
13, the anchor point is set up at (XO, YO) that is the center of the original image. It is 
also assumed that the directions of the coordinate axes are the same as the directions of 
the coordinate axes of the original image coordinate system. 

[0143] The anchor point is assumed to be fixed to the last of the animation. As a result, 
all values for the position and the rotation are decided on the coordinate system on 
which the anchor point is set. 

[0144] Next, the positions of the key frames and the move locus that connects the key 
frames are set up in the time axis. For example, as shown in Fig. 15, it is presupposed 
that the airplane is at point A in the first frame, then moves to point B after two seconds, 
and to point C after four seconds, and that the three points are connected with a curved 
line. Moreover, the rotation and the rate of enlargement and reduction are also set up to 
the three key frames that are the frames at the three points. A movement locus for a 
frame located between the key frames in the time axis is set up by the interpolation of 
the adjacent key frames. As a result, animation in which the airplane flies to the upper 
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right comer can be created as shown m Fig. 14. 
[0145] (Application Example 1) 

[0146] In the present example, after the position and range of the face part area are set 
up by template matching, the origin of the coordinate system (the anchor point) for the 
ornament is set at the point corresponding to the position of the face part area. 
[0147] Fig. 16 (a) shows an omament data. Fig. 16 (b) shows how the origin of the 
coordinate system (the anchor point) for the omament is set at the head part of the face 
part area. The omament depicts a partially elliptical orbit. Fig. 16 (c) shows the resultant 
composite image. 

[0148] When the person's face part area is detected, as shown in Fig. 16 (b), the anchor 
point is correspondingly moved to the head part of the face part area. Moreover, when 
the size of the face area changes, the scale of the coordinate system for the omament is 
changed in proportion to the size of the face area. 

[0149] When the direction of the face part area changes, the direction of the axis of the 
coordinate system is changed to the direction that is the same direction as in the first 
set-up. 

[0150] Thereby, even when a person moves relatively to a photographing equipment, 
the omament follows the movement of the person, always drawing elliptical orbit on the 
head part, 

[0151] Moreover, in a plurality of the face images photographed in different scenes, the 
omament can be moved on the orbit, which is always proportional to the size of the 
head part, corresponding to each face position, range, inclination, and direction. As a 
result, it becomes more interesting. 
[0152] (Application Example 2) 

[0153] In the present example, the omament information (part 2) is used, and the 
opacity of the omament in the composite image frame is changed following the 
movement of the face area. 
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[0154] The present example uses the ornament information (part 2) shown in Fig. 10 (b). 
The processing of Steps 1 to 3 in Fig. 5 is performed in the same manner as the case 
where the ornament information (part 1) shown in Fig. 10 (a) is used. 
[0155] However, the processing in Steps 4 to 6 is different. In Steps 4 to 6, using the 
mask of Fig. 12 (e), a still image shown in Fig. 12 (f) is visibly arranged in the shaded 
area, and the image detected as the face area within the input image is pasted in the 
unshaded area, after adjusting the size of the image of the face area to the size of the 
unshaded area. After the processing, the resultant composite image is outputted. 
[0156] Thus, the composite mask image with the still image as the background is used 
as the omament arrangement information. According to the result of the face detection, 
the center of the composite mask image is arranged at the center of the face. The set-up 
of the opacity is controlled by the composite mask image when the still image and the 
input image are composed. 

[0157] As a result, when composing with another image, the still image is prevented 
from overlapping with the face area. 
[0158] (Application Example 3) 

[0159] In the present example, as shown in Fig. 17, the depth information (Z-axis) 
regarding a viewpoint is added to both of the input image and the omament. 
[0160] When the omament is cut off in a virtual plane 60, which is equivalently the 
projection surface of the input image, the omament is divided into a foreground area 61 
that is located in the foreground side of the virtual plane 60, and a background area 62 
that is located at the background side. 

[0161] Furthermore, the opacity of the foreground area 61 and the opacity of the 
background area 62 are changed and the two areas are composed. Thereby, as shown in 
Fig. 18, the animation, which the star goes around the face, can be created. 
[0162] What is necessary is to set intentionally the anchor point so that the locus passes 
the face. By making the background 62 that passes through the back of the face 



21 



transparent, the ornament having the three-dimension-like motion can be added. 
[0163] (Application Example 4) 

[0164] In the present example, as shown in Fig. 19, the attribute (position and/or 
direction) of the ornament in the following frame is assumed by the attribute of the 
omament in the present frame and a series of the attribute in several previous frames. 
For example, if the omament is assumed to move at a constant and uniform speed, the 
attribute of the omament can easily be assumed by a simple equation. 
[0165] Furthermore, when the omament overlaps with the body part area, the orbit of 
the omament may be preferably changed. When plural omaments are made to appear, 
each omament is preferably allowed to move independently with each other. 
[0166] In the present example, as shown in Fig. 20, when the omament collides with the 
border line of the body part area or the input image, the moving direction of the 
omament is changed so as to move the omament away from the border line. Thereby, 
when the omament collides with the body part area, a reflection-like motion can be 
given to the omament. As a result, the composite image becomes more interesting. 
[0167] According to the present invention, even if both personal image and omament 
are moving, the omament can be designed not to interfere with the personal image, 
therefore, the personal image can be clearly displayed. Moreover, the composite image 
can be made more interesting by synchronizing the movement of the omament with the 
movement of the personal image. 

[0168] Having described preferred embodiments of the invention with reference to the 
accompanying drawings, it is to be understood that the invention is not limited to those 
precise embodiments, and that various changes and modifications may be effected 
therein by one skilled in the art without departing from the scope or spirit of the 
invention as defined in the appended claims. 
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